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LONG-TERM  GOALS 

In  FY  2007  the  Naval  Postgraduate  School  (NPS)  teamed  with  Virginia  Tech  (VT)  to  conduct 
fundamental  research  and  experimentation  aimed  at  enabling  coordinated  autonomous  operations  by 
teams  of  heterogeneous  (land,  sea,  air)  vehicles.  The  primary  goal  of  this  research  is  to  develop  tools 
for  vehicle-to-vehicle  cueing  and  distributed  sensing  for  shared  situational  awareness.  A  long-term 
goal  for  this  research  is  persistent  area  surveillance  of  harbor  and  riverine  environments  by  a 
“distributed  autonomous  system”  (DAS)  of  networked  vehicles  and  sensors.  Under  this  operational 
concept,  aerial  vehicles  would  function  as  long-range  sensors  and  provide  navigational  cues  to  surface 
vehicles  providing  close-up  video  inspection  or  interdiction  of  suspicious  vessels.  Sensor  data  analysis 
and  integration  would  provide  automated  scene  analysis,  object  detection  and  tracking,  and  vision- 
based  autonomous  navigation.  The  system  would  exploit  data  from  all  vehicles  and  sensors  to  produce 
three  dimensional  maps  of  the  operating  environment,  buildings,  ships,  etc. 

OBJECTIVES 

The  main  objective  of  this  project  was  to  develop  underlying  tools  and  technologies  needed  to  field 
and  demonstrate  coordinated  operations  with  a  team  of  autonomous  vehicles.  To  achieve  this  objective 
in  the  16-month  allotted  time,  the  project  team  focused  their  efforts  on  a  mutually  interesting  problem 
with  relevance  to  current  military  operations:  convoy  support  by  autonomous  vehicles.  The 
technologies  and  capabilities  developed  by  addressing  this  problem  are  necessary  steps  toward 
achieving  our  long-term  goals  in  follow-on  projects.  The  problem  was  decomposed  into  two  near-term 
project  objectives,  with  an  emphasis  on  experimentation. 

UAV-USV  cueing 

The  goal  of  this  project  is  to  develop  command  and  control  strategies  to  support  convoy  and  riverine 
operations  with  autonomous  vehicles.  The  concept  of  operations  (Figure  1)  involves  “tethering”  an 
unmanned  aerial  vehicle  (UAV)  to  a  ground  or  surface  vehicle  whose  sensor  range  or  maneuverability 
is  constrained  by  its  operating  environment.  The  unmanned  ground  or  surface  vehicle  (UGV  or  USV) 
commands  the  UAV  to  survey  the  area  ahead  of  its  intended  path.  Computer  vision  techniques  will 
enable  the  UAV  to  identify  navigational  hazards  (obstacles)  in  their  path  or  objects  requiring  closer 
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investigation  (targets).  The  UAV  would  then  provide  these  situational  awareness  cues  to  the  UGV  or 
USV  so  it  can  take  appropriate  action. 


Figure  1:  UAV-USV  Concept  of  Operations 


Sensor  Data  Analysis  and  Integration  for  Autonomous  Vehicles 

The  goal  of  this  project  is  to  automatically  analyze  sensor  data  from  unmanned  vehicles  and  provide 
data  visualization  for  enhanced  situational  awareness,  advanced  surveillance  capabilities,  and 
improved  vehicle  autonomy.  Vehicle  sensors  not  originally  intended  for  navigation  and  control  still 
provide  sufficient  and  often  superior  information  than  a  typical  GPS  and  inertial  measurement  unit 
(IMU)  can  supply.  Beyond  navigation,  these  sensors  (such  as  visible-light  cameras  or  LIDAR  devices) 
make  a  wealth  of  information  accessible  that  current  automated  analysis  methods  can  scarcely  tap  into. 
This  project  is  aimed  at  leveraging  more  of  this  information  through  automated  analysis  and  visual 
integration  to  make  steps  towards  achieving  our  long-term  goals 

APPROACH 

As  stated  above,  this  project  is  a  research  partnership  between  Virginia  Tech  and  the  Naval 
Postgraduate  School.  The  VT  team  was  led  by  Principal  Investigator  Dan  Stilwell  (USV  component), 
Craig  Woolsey  (Co-PI  and  UAV  component),  and  Charles  Reinholtz  (UGV  component).  This  report 
will  focus  on  work  conducted  by  the  NPS  team  led  by  Doug  Homer  (PI),  Isaac  Kaminer  (UAV 
component),  Mathias  Kolsch  (Computer  Vision),  and  Sean  Kragelund  (Project  Management,  USV 
component). 


UAV-USV  Cueing 

This  project  involved  several  subtasks  with  varying  degrees  of  coordination  with  our  VT  partners: 

•  Established  a  common  communications  architecture.  The  ITT  Mesh  Wireless  Modem  Card  was 
selected  as  the  primary  means  of  vehicle-to-vehicle  communications  for  this  project,  due  to  its 
proven  performance  and  reliability  on  NPS  UAVs  used  at  quarterly  Tactical  Network  Topology 
(TNT)  field  experiments. 

•  Implemented  standard  inter-vehicle  communications  protocols.  The  Joint  Architecture  for 
Unmanned  Systems  (JAUS)  was  selected  based  on  VT’s  expertise  and  their  JAUS-compliant 
UGVs.  A  common  protocol  was  necessary  for  VT  vehicles  to  interoperate  with  NPS  vehicles  at 
future  system  demonstrations.  JAUS  is  a  mature,  well-documented  standard  being  sponsored 
and  promoted  by  the  Department  of  Defense  (DoD) 

•  Selected  Wolfgang  Baer’s  Perspective  View  Nascent  Technology  (PVNT)  to  perform  target 
detection  and  localization  from  UAV  video.  PVNT  consists  of  a  user-operated  interface  and  a 
digital  terrain  database.  It  receives  UAV  telemetry  and  video  in  real-time,  and  generates  a 
synthetic  view  of  the  terrain  for  comparison  with  the  actual  video.  When  a  human  operator 
clicks  a  point  in  the  synthetic  view,  the  location  is  relayed  to  the  UAV.  In  this  way,  targets  can 
be  localized  to  within  the  accuracy  of  the  digital  terrain  data,  which  is  typically  better  than 
accuracies  achieved  by  low-cost  UAV  inertial  sensors. 

•  Established  a  development  and  experimentation  program  that  built  upon  our  most  mature 
vehicle  platforms  and  capitalized  on  existing  UAV  test  schedules  during  NPS  TNT  exercises. 
Our  system  baseline  consisted  of  the  VT  Rocky  UGV,  the  NPS  Rascal  UAV,  and  Wolfgang 
Baer’s  Persistent  View  Nascent  Technology  (PVNT)  software. 

•  Perform  land-based  testing  of  new  capabilities  before  transitioning  them  to  maritime  platforms. 

Sensor  Data  Analysis  and  Integration  for  Autonomous  Vehicles 

The  operational  and  technical  requirements  for  developing  this  technology  are: 

•  access  to  data,  both  live  and  stored  for  offline  processing  (repeatability),  both  vehicle  telemetry 
and  video,  including  time  stamps; 

•  data  analysis  capabilities,  mainly  a  set  of  image/video  processing  and  computer  vision  methods 
reflecting  recent  advances; 

•  powerful  graphics  capabilities,  both  regarding  the  rendering  hardware  and  high-performance, 
cutting-edge  software  packages; 

Time-synchronized  video  and  sufficient  sensor  telemetry  was  not  available  and  getting  this  data  from 
vehicles  in  real-time  requires  knowledge  of  their  communication  protocols.  Therefore,  we  addressed 
the  first  requirement  by  building  a  protocol  and  data  management  suite  for  autonomous  vehicles  called 
AVSuite.  It  facilitates  communication  among  autonomous  vehicles,  their  operators,  and  data  analysis 
stations.  It  also  permits  data  storage  and  replay  through  various  interfaces.  Our  approach  provides  for 
protocol  extensibility  to  accommodate  future  needs  and  changes. 

We  addressed  the  second  and  third  requirements  with  a  new  application  developed  for  this  project 
called  VideoTexture.  This  sensor  data  analysis  and  integration  tool  can  operate  on  either  real-time  or 
stored  mission  data.  It  facilitates  a  number  of  functions  ranging  from  video  stabilization  and 
mosaicing  to  user  interaction  with  large  three-dimensional  spatial  models.  Our  development  approach 
for  Video  Texture  used  the  following: 


•  Real-time  data  analysis  methods,  robust  model  estimators,  and  methods  designed  to  deal  with 
real-world  problems  such  as  video  transmission  noise; 

•  OpenCV  for  basic  computer  vision  functionality  as  well  as  portability; 

•  Advanced  OpenGL  features  for  good  rendering  performance; 

•  OpenSceneGraph  as  a  high-performance,  up-to-date  graphics  package  that  allowed  for  full 
customization  and  paged  levels  of  detail  which  are  required  for  rendering  large  areas  of  virtual 
terrain.  Furthermore,  OpenSceneGraph  is  based  on  OpenGL,  improving  portability. 

•  We  focused  on  (UAVs)  to  capture  data  that  was  visually  challenging,  yet  technically  feasible. 

WORK  COMPLETED 

Sensor  Data  Analysis  and  Integration  for  Autonomous  Vehicles 

AVSuite  is  an  extensible  and  portable  (cross-platform,  Windows,  Linux)  library  for  autonomous 
vehicle  data  management,  including  communication,  protocol  conversion,  and  data  storage.  This 
allows  multiple  autonomous  vehicles,  control  stations,  and  data  logging  stations  using  different 
protocols  to  communicate.  Adding  additional  inbound  and  outbound  protocols  to  the  library  is 
straightforward.  AVSuite  can  store  time-stamped  data,  including  video,  and  replay  data  logs  at 
varying  speeds.  Temporal  synchronization  between  telemetry  data  streams  and  video  is  built-in. 
Protocols  and  data  formats  currently  supported  by  AVSuite  include:  Cursor  on  Target  (CoT),  JAIJS, 
Predator  Exploitation  Support  Data  (ESD),  PVNT,  and  various  NPS  protocols  developed  to  support 
UAV  experimentation.  For  this  project,  rather  than  forcing  NPS  researchers  to  modify  their  protocols, 
we  used  AVSuite  to  translate  incoming/outgoing  JAUS  messages  to/from  their  expected  formats. 

VideoTexture  is  a  sensor  data  analysis  and  integration  tool.  It  takes  real-time  or  logged  mission  data 
and  video  as  inputs  and  performs  automated  analysis  and  data  visualization.  It  uses  “telemetry”  data, 
which  for  our  purposes  includes  aircraft  pose,  i.e.  position  and  orientation,  camera  position,  direction 
and  zoom  setting.  After  data  analysis,  it  displays  the  vehicle,  its  flight  path  and  user  annotations  in 
combination  with  the  virtual  terrain.  The  video  data  is  persistently  integrated  with  the  virtual  terrain  in 
real-time.  Its  ability  to  project  2D  video  onto  virtual  3D  terrain  in  real  time  creates  a  permanent 
mosaic,  enhancing  spatial  awareness  and  providing  a  means  of  updating  the  terrain  imagery  database. 
VideoTexture ’s  main  features  are: 

•  3D  UAV,  video,  and  topography  visualization 

•  6DOF  motion  and  perspective  projection  model  for  perspective  views  (trapezoidal  geo¬ 
rectification) 

•  improved  geo-registration  of  projected  video  through  frame-by-frame  mosaicing 

•  telemetry  data  correction 

•  relative  UAV/camera  pose  calculation  solely  based  on  video  (no  reliance  on  telemetry  data) 

•  digital  video  stabilization 

•  various  video  de-interlacing  modes 

•  robust  to  moving  and  changing  video  scenes  and  video  noise  (drop-frame  decision) 

•  satellite  image  (terrain)  can  be  permanently  updated  based  on  UAV  sensor  data 

•  graphical  user  interface  for  map  annotations,  object  tracking  selection,  and  road  overlays. 

•  symbols  conforming  with  MIL-STD-2525B 

•  all  features  are  available  at  real-time  processing  speeds 

•  platform  portability 


VideoTexture’s  mosaicing  and  geo-registration  capabilities  lay  the  groundwork  for  object  tracking 
through  visual  means,  further  aided  through  a  physical  motion  model.  We  will  also  improve  geo¬ 
registration  of  projected  video  through  frame-to-satellite  image  mosaicing.  VideoTexture’s  annotation 
capabilities  allow  users  to  mark  on  the  terrain  the  3D  positions  of  features,  paths,  and  regions  which 
can  be  designated  to  represent  items  like:  target  location,  target  search  region,  no-fly  zones,  and 
ingress  and  egress  routes.  Users  can  view  mission  video  in  real-time  or  playback  a  previous  mission 
together  with  a  video  mosaic  painted  onto  the  terrain  1)  as  the  mission  progresses,  or  2)  a  fraction  of 
the  entire  video.  VideoTexture  allows  interactive  examination,  annotation,  and  recording  of  analysis 
results  for  later  display  or  for  immediate  use  in  operational  mission  planning. 

RESULTS 

UAV-USV  Cueing 

The  work  for  this  portion  was  completed  in  stages  coinciding  with  the  quarterly  TNT  field  experiment 
schedule.  In  August  2006,  a  Virginia  Tech  team  traveled  to  Camp  Roberts,  CA  for  our  first  joint 
experiment.  The  goal  of  this  experiment  was  to  test  Mesh  point-to-point  communications  between  the 
NPS  Rascal  UAV  and  a  surrogate  UGV.  NPS  researchers  had  developed  a  Rascal  interface  whereby 
network  operators  could  command  the  UAV  waypoint,  orbit  and  camera  parameters  using  a  simple 
binary  protocol.  VT  researchers  implemented  this  protocol  with  their  UGV  control  software.  Using  a 
pickup  truck,  a  laptop,  and  the  inertial  navigation  system  (INS)  from  their  ground  vehicle,  we 
performed  a  simulated  UAV-UGV  operation.  The  laptop  took  control  of  the  UAV  and  proceeded  down 
a  narrow  roadway  with  limited  visibility  of  its  forward  path.  The  UAV  successfully  tracked  the  ground 
vehicle  and  an  artificial  obstacle  was  visible  in  the  UAV  video  before  it  could  be  seen  on  the  surrogate 
UGV  video  (Figure  2).  This  validated  our  concept  of  operations. 


Figure  2:  Initial  UAV-UGV  cueing  (surrogate  UGV)  experimental  data  and  aerial  video  still 


In  November  2006,  the  VT  team  shipped  the  VT  UGV  (“Rocky”)  to  Camp  Roberts  to  perform  a 
follow-up  experiment  with  the  actual  unmanned  vehicles  and  UAV  targeting  software  planned  for  our 
final  demonstration.  In  this  experiment,  the  UGV  tasked  the  UAV  to  continuously  monitor  a  point  on 
the  Rocky’s  preplanned  path.  As  Rocky  navigated  a  series  of  predefined  waypoints,  Rascal  provided 
situational  awareness  and  forward  surveillance  of  Rocky’s  intended  path.  Aerial  video  was  transmitted 
to  the  UAV  ground  station,  where  an  operator  using  GIS-based  Perspective  View  Nascent  Technology 


(PVNT)  software  identified  and  localized  an  obstacle  along  the  planned  UGV  route.  The  obstacle’s 
location  was  automatically  relayed  to  Rocky,  which  autonomously  modified  its  mission  to 
circumnavigate  the  point  and  perform  up-close  video  inspection  of  the  object  with  a  side-mounted 
camera  (Figure  3).  These  capabilities  were  transitioned  to  the  SeaFox  USV  and  field-tested  on  a 
reservoir  at  Fort  Hunter  Liggett,  CA.  in  April  and  May  prior  to  our  primary  system  demonstration  at 
Tyndall  AFB,  FL  during  AUV  Fest  2007. 


Figure  3:  Follow-up  UAV-UGV  cueing  experiment;  this  marked  the  first  time  that  the 
NPS  Rascal  UAV  was  controlled  by  another  autonomous  vehicle 


During  AUV  Fest  2007,  we  demonstrated  UAV-USV  cueing  between  Rascal  and  SeaFox  in  a  riverine 
environment.  Figure  4  shows  the  vehicle  positions  as  Rascal  operated  under  SeaFox  control.  A  forward 
observer  with  a  laptop  and  ITT  Mesh  card  running  PVNT  software  detected  and  localized  an  artificial 
target  (Figure  5).  This  data  was  relayed  to  SeaFox  via  the  Rascal  UAV.  At  this  time,  however,  SeaFox 
was  not  capable  of  autonomously  circumnavigating  a  target.  Instead,  it  treated  inspection  locations 
received  from  the  UAV  as  waypoints  to  be  followed.  In  July  2007,  we  implemented  new  navigation 
algorithms  based  on  artificial  potential  field  theory  to  provide  a  more  robust  target  inspection 
capability.  This  capability  was  demonstrated  in  August  2007  experiments  on  Monterey  Bay  and  Lake 
San  Antonio,  CA.  in  (Figure  6).  The  plot  on  the  left  shows  SeaFox  executing  a  pre-programmed  box 
pattern  when  a  network  user  sends  SeaFox  a  target  location.  SeaFox  transits  to  the  target  location, 
orbits  the  target  for  a  prescribed  time,  then  returns  to  the  loiter  box.  The  plot  on  the  right  shows 
SeaFox  orbiting  a  moving  target  after  a  network  user  provides  a  target  position,  speed,  and  heading. 


cueing  between  Rascal  and  SeaFox  at  AUV  Fest  2007 


Figure  4:  UAV-USV 


Figure  5:  Screen  shots  of  PVNT  image  mensuration  software  used  for  target 
detection  and  localization  from  NPS  Rascal  video  at  AUV  Fest  2007 


Path  Plot  from  08/28/D7  at  09:48:12 


Figure  6:  SeaFox  telemetry  plots  that  depict  new  target  inspection  behaviors.  At  left:  SeaFox 
departs  from  a  predefined  loiter  box  to  orbit  a  target  position  received  over  the  network.  At 
right:  SeaFox  orbiting  a  moving  target  when  cued  with  target  position,  range,  and  bearing 


Sensor  Data  Analysis  and  Integration  for  Autonomous  Vehicles 

AVSuite  and  VideoTexture  were  field-tested  at  several  exercises,  including  TNT  and  AUV  Fest  2007. 
Both  components  were  deployed  in  the  live  data  environment  and  captured  data  for  later  offline  use.  A 
typical  scenario  is  shown  in  (Figure  7). 


Figure  7:  A  sample  scenario  employing  AVSuite  and  VideoTexture 


The  UAV-USV  cueing  experiments  conducted  at  TNT  field  experiments  showcased  the  versatility  and 
usefulness  of  the  AVSuite  application.  AVSuite  was  successfully  installed  on  both  the  SeaFox  USV 
(Redhat  Linux  8.0)  and  the  Rascal  UAV  (Windows  XP).  Each  vehicle  ran  two  instances  of  the 
program  to  translate  inter-vehicle  messages  to/from  standardized  JAUS  messages  (Figure  8).  These 
experiments  were  crucial  to  demonstrating  UAV-USV  cueing  within  the  JAUS  communications 
framework  chosen  for  this  project.  As  a  result,  the  NPS  vehicles  achieved  JAUS  Level  One 
Interoperability,  and  were  able  to  communicate  with  Virginia  Tech’s  vehicles  at  AUV  Fest  2007. 


Rascal  Mesh  Computer 
Windows  XP 
192.168.112.6 


SeaFox  Secondary  Controller 
Redhat  8.0  (Linux) 
192.168.112.53 


Figure  8:  AVSuite  used  as  a  Protocol  Translator  for  the  Rascal-SeaFox  cueing  demonstration 


All  VideoTexture  features  described  above  were  successfully  tested.  Figure  9  shows  3D  Camp 
Roberts  terrain  with  the  video  projected  onto  it.  Figure  10  shows  more  detailed  snapshots  of  terrain 
with  elevation  and  the  (brown-grey)  video  overlaid.  Figure  11  demonstrates  the  real-time  video 
mosaicing  capability  which  corrects  the  telemetry  data  based  on  video  sensor  measurements  (feature- 
based  tracking  and  self-localization). 


Figure  9:  Camp  Roberts  and  UAV  video.  The  UAV  “projects”  its  video  onto  the  3D  terrain. 


Figure  10:  Two  UAV  positions  (on  the  left  and  right)  and  two  view  angles  of  the  respective  video 

onto  the  terrain  (top  and  bottom). 


Figure  11:  Sequence  of  images  showing  how  a  video  mosaic  is  painted  onto  terrain  data. 


Figure  12  shows  some  of  the  annotation  capabilities  of  the  VideoTexture  application.  The  user  has 
marked  a  road  with  “geo- stationary”  markers.  The  right  image  shows  the  same  information  in  a  more 
familiar  2D  view,  also  using  MIL-STD-2525B  icons. 


Figure  12:  Terrain  annotation  with  the  VideoTexture  application,  3D  view  on  the  left 

and  2D  map  view  on  the  right. 


RELATED  PROJECTS 

The  work  of  our  Virginia  Tech  research  partners  is  discussed  in  a  project  entitled  “Coordinated 
Sensing  and  Control  for  Surveillance  and  Tracking  by  Heterogeneous  Autonomous  Vehicle  Teams.” 
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